The future of Search: context aware, local, and mobile with augmented reality

Following four extraordinarily compelling hours of interviews to the Googleplex, with Google Fellows and lead engineers Amit Singhal, Matt Cutts, Ben Gomes and Trystan Upstill – covering the emotional through to the arcane facets of the future of search – I wrote the following article with my colleague Mike Hanley. It was published in the April 2010 edition of The Australian Financial Review’s Boss Magazine.

Hitchhiker’s guide to the Googleplex. In a complex world, search is the answer. But the question is: where is it all heading?
There was a power cut when Amit Singhal was getting ready to drop his children off at school on Wednesday, February 17. He reached for his mobile phone and typed in “power cut Palo Alto”. In a microsecond, he could see that people were tweeting about a plane crash that had cut the power lines to his town. News reports followed. Three auto executives had been killed, and a pall of tragedy had been cast over Singhal’s morning.

For this man, that coincidence of fate – the butterfly of the tragedy causing the hurricane of the ruined morning – is as good a metaphor as any for the future of search. And Singhal should know, for he has done more than just about anybody on the planet to make it happen. His job title is Google fellow, and he has worked on the science of search since before the World Wide Web existed. As the man responsible for it, he knows more about the insides of the Google engine, its arcane and complex mathematics, its tics and tricks, its formulas and workarounds, its algorithm, than anyone else. Even via video-conference from California, Singhal radiates gratitude for a world so complex that it makes the science of search possible.

For Singhal and his engineering colleagues, search is all about how the world fits together, the miniscule idiosyncrasies that describe the shortest mathematical distance between you and the restaurant you are looking for. Not simply the restaurant that matches the words you typed into the search engine, but the closest match between the ideal restaurant that would give you the most happiness and what is available in the world.

To do this, the search scientists need to know three things: what’s out there, where you are, and what’s inside your mind. They have done a lot of work on the first two, and are making lots of progress on the last, but remarkably most reckon they are no more than 10 per cent towards uncovering the complete universe of search. The question is, if it is already this easy to find the restaurant that I typed in, what could we do if we were 50 or 70 per cent along the path?

“I think the question that you would ask at that point is: What can’t you do with search?” Singhal says. “Already you can find information on what happened seconds ago, as the Palo Alto plane crash example shows. That would not have been possible even last year – computers wouldn’t have worked because the power was out, and phones weren’t powerful enough. In the future, search will be the technology that not only answers your information needs but also integrates with all other technologies, so that you will know whatever you need to know, whenever you need to know it, on whatever device you need it.”

Search is already the platform that connects Singhal’s home life with the plane crash. It is also the technology that notifies Stanford Hospital, also affected by the power cuts, to activate its contingency plan and order ice for its refrigerators to cover the expected power outage hours.

Search is already deeply embedded in our lives, not only through the ubiquitous Google, but also on every website, on your computer’s desktop and on your mobile and iPod. But Singhal is looking to the time when we won’t even call it a search experience – it’ll just be our life experience.

Here are a few for instances:
You are standing on the corner of Bourke and Elizabeth streets in Melbourne, and a bit thirsty. You pick up your mobile device and say, “thirsty”. The search engine knows where you are, the proximity of convenience stores and juice bars around you, what they have in stock, their prices and – here’s the kicker – it understands that you’re probably a bit thirsty because you had an intense workout this morning (it has access to your calendar) and are, most likely, mildly depleted of minerals (you’ve told it about your health condition). Your device gives you several options, directions and prices, and recommends a nutritionally appropriate solution for your thirst problem.

You are in a foreign city. You use your mobile device as a virtual tour guide by pointing it at buildings so that it provides you with historical details, interesting facts and suggestions for things to do. One of these suggestions is to meet with an old school friend, who happens to be in the neighbourhood. The search engine recommends a nearby restaurant that matches both your tastes and hers.

You are undertaking some arcane research for a lecture on Rene Descartes. Using textual analysis, your search engine identifies a letter dated May 27, 1641, signed by the philosopher, never previously discovered. (This actually happened to Dutch philosopher Erik-Jan Bos in January this year.)

The innovation machine
Novelist William Gibson said: “The future is already here, it’s just unevenly distributed”. These scenarios would have seemed like science fiction just a few years ago but the technologies behind them are already being stitched into our lives. The world of search is not a completely open field for Google; companies such as Microsoft’s Bing, Wolfram | Alpha and even Facebook are coming at the problem from different angles, but Google is at the heart of the drive to create a future of “ambient findability”, where we can find anyone or anything from anywhere at anytime.

At the heart of that effort lies the company’s secret sauce, its algorithm, the hugely complex and mysterious series of mathematical equations that determines which of the infinite number of web pages out there will pop up first when you make a query.

Matt Cutts, a senior engineer at Google and, at one point, the mysteriously faceless face of Google known as the Google Guy, likens the algorithm to a complex machine: “People always ask: ‘Is there one algorithm? One monolithic algorithm that everything feeds into?’ That’s like asking, ‘Is a car a machine?’ Yes, a car is clearly a machine, but a car is also composed of many different machines. You’ve got the carburettor, then even within the carburettor you’ve got the pistons. So I think of Google like a car – maybe it’s more like a space shuttle – as a device that propels you forward and gives you understanding. But it’s not one monolithic machine, it’s made up of many smaller parts.”

Last year, Google made about 500 changes to the algorithm. Each week, the launch committee, a team of Google’s top engineers, meets to discuss the ideas being put up to improve how the “space shuttle” flies. At each meeting, the team con- siders 10 or so ideas, each of which has been tested within the company’s “sandbox” – a parallel Googleverse within which the company’s engineers are free to try experiments. They can add their proposed modifications to the algorithm and see whether it makes the world a better place or not.

Launch committee meetings are jovial affairs, Singhal says: “We run this very openly, they’re all there, it’s a conversation, there’s nothing heavy handed about it. From a management perspective you can’t afford to make the cost of perceived failure too high. It’s really very informal, everybody is joking around. … We talk about all sorts of things, and everyone feels very comfortable. Most of the ideas that come there receive one of two feedbacks: ‘Great, let’s do it’, or ‘Hey, if you tried XYZ, wouldn’t that make it much better?’ ”

On the user experience side, at any one time, Google is undertaking numerous experiments with its interface – that plain white page we all know so well. You may have noticed lately that when you start typing a query, Google now suggests to you what you might be thinking of. Before that was introduced, Google tested it on a subset of perhaps a million users, measuring performance and its impact on results all the while, just as it does with all its changes to its closely watched user interface. Engineers, led by user experience guru Ben Gomes, look at metrics to determine if queries come up quicker; if users click on the first one or two results, or if they have to click through to the second or third pages; if they have to go back and refine their query; if they misspell things and the engine serves up the correct alternative. All of this to determine whether, in their eyes, the world is a better place before or after the innovation.

Bing
When Microsoft launched its new search engine, Bing, in 2009, the marketing strategy was to emphasise that it isn’t a search engine, it is a decision engine. this conceptual shift indicates where the science of search is headed – towards comprehending context. the assumption is that people search for things for a reason, to reach some kind of decision. We are looking for answers to specific issues that will enable us to make a judgement, for instance, where to buy a digital camera, at the best price, closest to our home, with which features, and from which manufacturer. Bing’s ambitions may be in the decision-engine space, but for the moment, it looks and feels just like a search engine, with its results pages eerily similar to those of google. it does come up with different results, because it uses a different algorithm, but to the untrained eye it’s difficult to see rhyme or reason. Websites such as Bing & Google provide comparative results.

Wolfram | Alpha
Wolfram | Alpha, also launched last year, is a computational knowledge engine, rather than a search engine. unlike google or Bing, Wolfram | alpha serves up only rationally knowable facts, so searching on “digital camera” will supply only a dictionary definition. it is, however, able to answer questions, such as “fat content in chips” (average 7.7 grams per 30 grams of potato crisps); “apple versus iBM share performance” (apple outperformed iBM by a factor of more than two over the last 12 months), or “population of new york versus the population of Sydney” (new york is 90.32 per cent larger). to achieve this level of statistics and computation, it has an engine powered by 5 million lines of computer code that require more than 10,000 computer chips to run. the computational algorithms it uses have a basis in artificial intelligence and its aim is “to collect and curate all objective data; implement every known model, method and algorithm; and make it possible to compute whatever can be computed about anything”. last year, Bing entered into partnership with Wolfram | alpha to incorporate its computational results into its search capabilities.

Question of trust
For engineers, the world may be complex, but the ethics of search are simple. Make it work better and you have improved the world. Would that it were so for all of us.

For business, Google’s dominance means that being found by the search engine can make or break companies. Search engine optimisation has become a science in itself, with SEO professionals constantly chasing a top ranking – a goal that is constantly moving, not least because Google is always tweaking its inner workings. Even for the smallest businesses, it is no longer feasible to structure a marketing strategy that does not include a budget for Google AdWords.

The company’s market dominance has sparked an inquiry by the European Commission into the inner workings of the search giant. Tangling with Google is something that not even the Chinese government can do without global repercussions; when it was found by the company to be hacking the Gmail accounts of supposed dissidents, Google took it on, with the might of the US government behind it.

Both these cases speak directly to the issue of trust. Customers need to be able to trust the search giant to produce results independent of its commercial interests. What’s more, they need to know it is keeping its ever-increasing cloud of Gmail and Google documents free from the prying eyes of governments and corporations, including itself.

Singhal and his colleagues talk blithely of a future of personalised search, context-relevant results and understanding what you want, when you want it. But many of us don’t want Google to know what we want. An engineering solution for this is for users to elect to provide data or not. But for many, this will not be safeguard enough. Privacy issues come last in the world of the engineer, or at least, they come second to developing the technologies that might abuse it.

For the moment, though, the engineers are running to stand still. The universe of information on the internet is expanding faster than the science of search can keep up. For instance, Twitter exploded in popularity just 12 months ago, and is now a large component of “real-time search”. There are millions of web pages out there that create useful data but need input to do so – for example, sites that demand a postcode before delivering up local service information. Google is developing ways of extracting the data from those sites to make it available through search. There’s also the Tower of Babel problem: if you want to deliver the world’s information, you have to do it in the language your users speak, so Google is hammering away at translation tools. Books, scholarly articles, videos, musical lyrics – each of these provides a rich repository of knowledge that ought to be searchable, in Google’s eyes, so its engineers are sicced on the problems standing in the way.

In the end, what Google is searching for is what we’re all searching for: meaning. In the world of search, this starts with understanding that when you type in GM foods, you’re looking for genetically modified foods, but when you type in GM cars, you’re after General Motors. But it ends with maximising the experiences that make up your life by connecting you, as directly as mathematically possible, to a universe of information. “Twenty years ago, I never dreamed we’d be able to do what we can today,” Singhal says. “But it is nothing like the systems that we will build in the future.”